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What is claimed is: 

1 . A method for crawling for resources in a network, the method comprising: 

receiving a list of resources on the network and for at least one of the resources on 
the list of resources, 

sending a first request to a server in the network for the resource using a 
first browser state, and 

sending a second request for the same resource using a second browser 

state. 

2. The method of claim 1 wherein the resources are identified by uniform resource 
locators and wherein the first and second request specify a same uniform resource 
locator. 

3. The method of claim 1 wherein the browser state comprises a language preference. 

4. The method of claim 1 wherein the browser state comprises a locale preference. 

5. The method of claim 1 wherein the browser state comprises location information. 

6. The method of claim 1 wherein the browser state comprises a browser identification. 

7. The method of claim 1 wherein the browser state comprises a network address. 

8. The method of claim 1 wherein the first request and the second request are issued by 
a first and second crawler applications that respectively have a first and second browser 
state. 

9. The method of claim 1 wherein the first and second requests are issued by a crawler 
application that varies its browser state between the first and second requests. 
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10. A method for processing crawled resources in a network, the method comprising: 

receiving a resource in response to a request for the resource using one of a 
plurality of browser states; 

storing the resource; and 

indexing the resource, the indexing step further comprising the step of associating 
the resource with a first browser state where the first browser state is the one of the 
plurality of browser states used to request the resource. 

1 1 . The method of claim 10 wherein resources are identified by uniform resource 
locators and wherein at least a first resource and a second resource identified by a same 
uniform resource locator are associated with different browser states. 

12. The method of claim 1 1 wherein the first and second resources are both stored only 
if the second resource is different from the first resource. 

13. The method of claim 12 wherein if the second resource is a duplicate of the first 
resource, a reference is stored that associates the stored first resource with the second 
browser state. 

14. The method of claim 10 wherein the browser state comprises any one of a group 
consisting of language preference, locale preference, location information, browser 
identification, and network address. 

15. A method for searching a database of crawled resources, the method comprising the 
steps of: 

receiving a search query from a browser client; 
detecting a browser state for the browser client; and 

searching for results from the database of resource using both the search query 
and the browser state of the browser client. 
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16. The method of claim 15 wherein the database includes at least one record which 
associates a first resource and a second resource in the database with a same uniform 
resource locator but with different browser states. 

17. The method of claim 15 wherein results that match the search query are filtered 
using the browser state of the browser client. 

18. The method of claim 1 5 wherein a relevance function is utilized to rank results from 
search of the database and wherein the relevance function considers the browser state of 
the browser client in ranking the results. 

19. The method of claim 15 wherein if the browser state of the browser client does not 
match any of the browser states in the database, then a default browser state is used in the 
search. 

20. The method of claim 1 5 wherein the browser state comprises any one of a group 
consisting of language preference, locale preference, location information, browser 
identification, and network address. 

21 . A computer-readable medium comprising one or more instructions which when 
executed perform the following: 

receiving a list of resources on the network and for at least one of the resources on 
the list of resources, 

sending a first request to a server in the network for a resource using a first 
browser state, and 

sending a second request for the same resource using a second browser 

state. 
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22. The computer-readable medium of claim 21 wherein the resources are identified by 
a uniform resource locator and wherein the first and second request specify a same 
uniform resource locator. 

23. The computer-readable medium of claim 21 wherein the browser state comprises 
any one of a group consisting of language preference, locale preference, location 
information, browser identification, and network address. 

24. A computer-readable medium comprising one or more instructions which when 
executed perform the following: 

receiving a resource in response to a request for the resource using one of a 
plurality of browser states; 

storing the resource; and 

indexing the resource, the indexing step further comprising the step of associating 
the resource with a first browser state where the first browser state is the one of the 
plurality of browser states used to request the resource. 

25. The computer-readable medium of claim 24 wherein resources are identified by 
uniform resource locators and wherein at least a first resource and a second resource 
identified by a same uniform resource locator are associated with different browser states. 

26. The computer-readable medium of claim 25 wherein the first and second resources 
are both stored only if the second resource is different from the first resource. 

27. The computer-readable medium of claim 26 wherein if the second resource is a 
duplicate of the first resource, a reference is stored that associates the stored first resource 
with the second browser state. 

28. The computer-readable medium of claim 24 wherein the browser state comprises 
any one of a group consisting of language preference, locale preference, location 
information, browser identification, and network address. 
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29. A computer-readable medium comprising one or more instructions which when 
executed perform the following: 

receiving a search query from a browser client; 
detecting a browser state for the browser client; and 

searching for results from the database of resource using both the search query 
and the browser state of the browser client. 

30. The computer-readable medium of claim 29 wherein the database includes at least 
one record which associates a first resource and a second resource in the database with a 
same uniform resource locator but with different browser states. 

3 1 . The computer-readable medium of claim 29 wherein results that match the search 
query are filtered using the browser state of the browser client. 

32. The computer-readable medium of claim 29 wherein a relevance function is utilized 
to rank results from search of the database and wherein the relevance function considers 
the browser state of the browser client in ranking the results. 

33. The computer-readable medium of claim 29 wherein if the browser state of the 
browser client does not match any of the browser states in the database, then a default 
browser state is used in the search. 

34. The computer-readable medium of claim 29 wherein the browser state comprises 
any one of a group consisting of language preference, locale preference, location 
information, browser identification, and network address. 
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